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ABSTRACT 

This paper asserts that rigorous research on the impacts of 
social programs to improve children’s prospects are necessary but not 
sufficient, because too many programs are multidimensional, cannot or should 
not be standardized, evolve or adapt through time, require participants' 
active involvement, or are heavily dependent for success on good 
implementation, not just good design. It recommends more flexible forms of 
evaluation or assessment that require experts and practitioners to 
hypothesize linkages between actions and outcomes, prioritize actions, 
identify interim indicators of success, and pay more attention to program 
attributes and institutional contexts that are essential to success. While 
recognizing the trade-off between knowing a few things very well and more 
things with less certainty, the paper argues for greater efforts to 
understand broad patterns that connect activities to results, even if this 
means compromising the search for absolute truths. Five appendices present: 
characteristics of effective interventions that make them difficult to 
evaluate with prevailing methods; examples of actions that contribute to 
school readiness; examples of indicators that measure progress toward sub- 
goals and toward outcome of school readiness; examples of attributes of 
effective services and supports; and examples of community and systems 
contexts that contribute to effectiveness. (Contains 25 references.) (SM) 
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PREFACE 



Random assignment studies of social programs have become the “gold standard” or sine 
qua non of researchers interested in determining what works to improve children’s prospects. In 
this paper, Lisbeth Schorr, a member of the Children’s Roundtable and Director of the Pathways 
Mapping Initiative of the Project on Effective Interventions at Harvard University, asserts that 
rigorous research on impacts is needed but is not sufficient. Too many programs are 
multidimensional, cannot or should not be standardized, evolve or adapt through time, require 
participants’ active involvement, or are heavily dependent for success on good implementation, 
not jxist good design. 

Schorr argues for more flexible forms of evaluation or assessment that require experts 
and practitioners to hypothesize the linkages between actions and outcomes, prioritize actions, 
identify inte rim indicators of success, and pay more attention to the attributes of programs and 
the institutional contexts that are essential to success. While recognizing the inevitable trade-off 
between knowing a few things very well and more things with less certainty, Schorr argues for 
greater efforts to understand broad patterns that connect activities to results, even if this means 
compromising the search for absolute truths. 



Isabel V. Sawhill 

Director, Children’s Roundtable 

The Brookings Institution 
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Determining “What Works” in Social Programs and Social Policies: 

Toward a More Inclusive Knowledge Base 

Lisbeth B. Schorr 

Even in this era of growing cynicism and shriveling resources, local groups in many parts 
of the country are determined to mobilize political will around achieving the tangible outcomes 
that the public values. Individuals and organizations are forming coalitions and acting together 
because they have found that fragmented, piecemeal, narrowly categorical approaches do not 
achieve their goals. 

But these groups are often stymied when they search for reliable and coherent 
information about “what works” in order to improve specific outcomes. Much of the readily 
available guidance focuses on the difficult processes of building partnerships, engaging 
neighborhood residents, and developing new forms of governance. This is valuable information, 
but very little guidance is available on the content of effective interventions. Community 
coalitions may find agencies and interest groups that make a persuasive case for channeling 
resources in their direction. Or they may be told, “Nobody knows what will work here. You 
have to figure it out for yourself, because each community is unique,” as though having 
information about what has worked elsewhere would deny each co mm u ni ty’s uniqueness. If 
they are persistent and sophisticated about where to look, community groups may find a 
publication or policy analyst or academic source that tells them, “Yes, a few reliable studies have 
indeed been done that document ‘what works,’ and these studies suggest a small number of 
proven programs that you may want to try to replicate.” 
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The lack of a broad range of solid, readily accessible, coherent information about “what 
works” to change community-wide outcomes is frustrating, not only to local activists, but also to 
service providers, policy-makers, and fimders. Those who have sought a few simple solutions 
that can be defined and mandated from above have found that models parachuted into local 
communities rarely succeed in the way the original, elegantly proven pilot project did. 
Replications of successful programs may turn out to be diluted or distorted and without much 
impact. Most funders, public and private, know that messy implementation efforts often 
sabotage the best-laid designs. Even as they recognize that advocates who can rally political will 
around a simple, unitary answer to a persistent problem get the most attention, they worry about 
how short-lived the single-pronged, narrowly bounded solutions seem to be. 

At the same time, community collaboratives, program designers, practitioners, and 
fimders are striving to get away from ill-considered fads and well-intentioned efforts that lack 
real impact. As they attempt to encourage and engage in more rigorous thinking about reform, 
they need more help than they are now getting from researchers and evaluators. It is the thesis of 
this paper that, to produce the broad knowledge base now needed to promote, develop, and 
spread effective strategies that will actually improve lives: 

• Prevailing conventions about what counts as credible knowledge must be re-examined 
and modified. 

• Our focus must change from making yes/no judgments about individual interventions 
to discerning patterns from an accumulation of research and experiences. 

• Our knowledge-building activities must cross systems and disciplinary boundaries, 
even if service delivery and fimding continue to operate predominantly within self- 
contained silos. 
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Prevailing Approaches to Determining “What Works” 



Conventional methods of assessing “what works” have provided information primarily 
about interventions where a single circumscribed problem is addressed by a single circumscribed 
remedy. But, as social problems have become deeper and more complex over the last 20 years, 
interventions that could respond to the growing burden of risk have also become increasingly 
complex — and therefore harder to assess by traditional means. 

Interventions that change only one thing at a time often fail because they change only one 
thing at a time. Many of the change initiatives that now seem most promising are complex 
efforts with multiple, interacting components that require constant mid-course correction and the 
active engagement of committed human beings. This applies equally to initiatives aimed at 
strengthening families, rebuilding neighborhoods, reforming schools, enhancing youth 
development, and assuring that disadvantaged four- and five-year olds are ready for school. 

In addition, promising interventions typically involve continuous adaptation to local 
needs and strengths, to lessons learned, and to changing circumstances. They are deeply 
dependent on context. ' Other factors add to the mismatch between what may be the “gold 
standard” of effective complex interventions and what is often referred to as the “gold standard” 
of evaluation methodology. Many leaders of exemplary programs are reluctant to participate in 
experiments involving random assignment of participants because they consider an individual’s 
or family’s ability to exercise choice crucial to the effectiveness of an intervention. Or they may 
be unwilling to shuffle children and families fi-om one ‘condition’ to another as though they were 
seeds m an agricultural experiment. 

* National Research Council (NRC) (2002); Berliner (2002); Schorr (1988 and 1997) 

^ Gardner (2002). 



ERIC 



6 



5 



These characteristics make it hard to use experimental methodology to assess the most 
promising efforts to unprove important outcomes. Perhaps even more damaging, program 
design itself is distorted by the push for proof, often distorting programs even before they get off 
the ground. Throughout the last two decades, evaluators have been telling program people that if 
their program did not fit into the Procrustean evaluation bed, they should simplify, narrow, 
standardize, or otherwise change their design. (For a summary of the characteristics of many 
promising interventions that are poorly matched with experimental approaches to evaluation, see 
Appendix 1). 

Of course, interventions that have components that can be evaluated through randomized 
experiments, without undermining their effectiveness, should be. There is no substitute for the 
certainty that attaches to experimental findings. But it is the very nature of the most promising 
responses to persistent social problems that makes them almost impossible to evaluate by the 
methodologically elegant ways in which we evaluate drugs or electric toothbrushes. So 
traditional evaluation research has excluded from consideration precisely those interventions that 
are most likely to have an impact. This fact, along with the tendency to evaluate programs long 
before they are “proud”^ and regardless of whether they are well implemented, helps to explain 
how Senator Daniel Patrick Moyruhan came to embrace “Peter Rossi’s Law,” which holds “that 
the expected value for any measured effect of a social program is zero.”^ 



3 

Nevertheless, pressure to use random assignment experiments to assess complex interventions is growing. In 
education reform, the current political focus on a narrow concept of “scientific research” as the basis for action 
seems to assume that virtually the only research that is credible is experimental. (Reeves, 2002, Viadero, 2003). A 
report from a Brookings Institution meeting on education reform suggested that proposed education reforms should 
be judged legitimate or illegitimate by the application of randomized field trials in which, “One person gets the pill 
and the other person gets the placebo.” (Peterson, 1999). 

^ “Proud” is the term applied by evaluation guru Donald Campbell to programs that are stable and based on well- 
developed strategies, that have been solidly implemented over period of years, and which can “send up a flag that 
says ‘we’ve got something special that we know works and think others ought to borrow.” Campbell (1993). 

’ Offiier (2001). 
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When policy makers or academics base their judgment of effectiveness - as they often do 
- primarily on the elegance of the evaluation methodology, many promising intervention 
strategies never make it into contention. Only very slim pickings emerge when the 
methodological funnel is constructed as narrowly as it typically is today. ^ Thus society is robbed 
of rich information about interventions whose causal connections have not been proven but from 
which we desperately need to learn to make wiser decisions about resource allocation and 
program design. 

Alternative Approaches to Determining “What Works” 

In the face of the many difficulties in determining “what works” in complex 
interventions, some participants and observers of community-based efforts have simply given up 
on all impact evaluation, and look to richly detailed, qualitative, descriptive methods to provide 
needed documentation. But these can perform only a limited function. While case studies and 
deeply contextualized descriptions of efforts to bring about change are a wonderful source of 
inspiration and information, when they stand alone they are very hard to generalize fi-om. When 
they make no attempt to provide even common sense connections between actions and results, 
when they do not stipulate even a hypothesized theory of change, they contribute little to a better 
understanding of how improved outcomes are best achieved. Given the current push for 



^ Two examples: The Collaborative Center for Child Health and Development, charged with discerning connections 
between early childhood interventions and adult success, commissioned researchers around the country to examine 
more than 2,000 published, peer-reviewed studies of early childhood development, but found only 17 [sic] of these 
of sufficient methodological rigor to use in coming to their conclusions. 

In a 2002 review for the Brookings Roundtable on Children of “what works” in after-school and youth 
development programs, economist Robinson Hollister found only 10 programs that met his criteria for rigor of 
design and therefore worth replicating or scaling up. 
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accountability, descriptive information is a welcome supplement to, but no substitute for, 
outcomes-based information. 

But in the last decade there have been a number of significant efforts to develop 
alternative approaches to assessing the impacts of complex interventions, especially when these 
are intended to bring about community-wide or institutional change. These approaches move 
away fi'om relying on randomized experiments as the single best evaluation methodology and 
seek to find ways of matching study designs more rigorously with the research questions they are 
intended to elucidate.^ 

The Manpower Demonstration Research Corporation and Public/Private Ventures have 
been especially creative in developing evaluation research designs to shed light on urgent policy 
and programmatic questions. They have modified random assignment methodology in ways that 
would allow them to obtain impact information from complex and non-standardized 
interventions.^ Additional alternate approaches include those taken by evaluators and analysts 
using Theory of Change Evaluations,^ and by some who systematically attempt to rule out “rival 
hypotheses” - explanations for changed outcomes other than the program or strategy in 
question. 

Another interesting variation, known as design-research, has evolved over the last 1 0 
years in the education domain. It is receiving greater attention in response to current concerns 
that education research should be more “scientific.”* * * Design-research focuses on research in 
real-world situations; “in order to find out what works in practice,” it gives up on the notion of 



^ Shavelson (2003). 

* For examples, see Bloom (2002), Brock (2002), Grossman (2002). 

’ Connell and Kubisch (1998); Weiss (2002). 

Yin (2000). 

’ ’ The role of design experiments in education research is the topic of the January/February 2003 issue of the 
publication of the American Educational Research Association. 
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controlling variables. By blending “empirical educational research with theory-driven design 
of learning environments,”*^ design research makes possible generalizations that depend more 
“on an interpretive framework than on sampling. ”*‘* 

Some of the interest in alternative methods seems to be based on the belief that only a 
stop-gap solution is needed. The hope is that the problem is a temporary one, that the research 
community simply needs to get around to doing the experiments that will provide the 
authoritative answers the public seeks. The idea that the missing definitive research is just 
around the comer is a theme that pervades From Neurons to Neighborhoods , the groundbreaking 
synthesis of what is now known about early child development, produced by a prestigious 
committee of the National Academy of Sciences. *^ In its discussion of intervention research, the 
report refers frequently to a science that “remains to be developed,” and to systematic research 
that has “not yet been done.”’^ It may be, however, that traditional research methods will never 
be able to provide all the definitive answers the committee is hoping for. Rather, as the report 
ultimately concludes, effective service delivery and in formed policy making will have to rest on 
the ability to make reasonable judgments and avoid irresponsible practices in the face of 
incomplete knowledge. * ^ 

Among those who see the problem as neither temporary nor superficial are many who are 
shifting their focus from assessing the effects of individual programs or strategies, and are 
attempting to draw inferences from multiple sources of evidence, analyzed in the context of 
sturdy theory. * * They agree with the observations of educator/ psychologist Jerome Bruner, that 

Collins (1999). 

Design-Based Research Collective (2003). 

Shavelson (2003). 

National Research Council and Institute of Medicine (2000). 

“ NRC (2000, pp. 370 and 358). 

'^NRC (2000, p. 411). 

Sampson (1998). 
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“plausible interpretations (are) preferable to causal explanations, particularly when the 
achievement of a causal explanation forces us to artificialize what we are studying to a point 
almost beyond recognition as representative of life. 

The next two sections of this paper describes one such effort, the Pathways Mapping 
Initiative, to assemble a knowledge base by applying reasonable judgments and plausible 
interpretations to “a preponderance of evidence coupled with strong theory.”^® 



The “Mental Mapping” Approach of the Pathways Mapping Initiative 

The Pathways Mapping Initiative (PMI) seeks to give communities a broad, deep, and 
coherent body of information that can be used to strengthen efforts to improve community-wide 
outcomes. PMI was established in January 2000 as part of the Project on Effective Interventions 
at Harvard University, with support from the Annie E. Casey Foundation. It aims to build on the 
wealth of findings now available about “what works,” going beyond both anecdotes and the 
traditional evaluation literature. 

PMI’s work is based on the conviction that communities, funders, and policy-makers 
should not have to start with a blank slate or to scrounge, unaided, to uncover the rich lessons 
learned by others. We believe that communities will be able to act most effectively when they 
can combine local wisdom and their understanding of local circumstances with accumulated 
knowledge, drawn from research, theory, and practice, about what has worked elsewhere, what is 
working now, and what appears promising. 

Although the nation has learned so much in the last decade about how families, 
neighbors, and social institutions can improve the life chances of children growing up in 

Bruner (1990) 

Sampson (1998). 
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America’s tough neighborhoods, most of the information that is readily available is not well 
matched to the needs of communities because it: 



• comes in small, isolated, and disjointed pieces; 

• arrives too late; 

• is derived from a severely limited range of interventions sufficiently circumscribed 
and standardized to allow for elegant evaluation; and 

• rarely identifies the essentials of what made the intervention work (and therefore 
cannot protect against the dilution and distortion that occurs when programs move 
from the hothouse conditions that produced strong evaluation results to the real-world 
conditions of scale-up - especially when scaling up involves public money). 

By trying to bring breadth, depth, and coherence to a world of social policies and 
programs that has grown increasingly fragmented as it has become more complicated, the 
Pathways Mapping Initiative is constructing a knowledge base with the specific intention of 
making it useful to communities. 

The PMI approach to assembling and distilling knowledge does not limit its explorations 
to identifying individual programs whose success has been proven; it casts a wider net to 
assemble evidence on “what works.” In our efforts to expand the search for - and inform the 
design of - social policies, programs, and strategies that will lead to improvements in specified 
outcomes, we have relied primarily on a consensus process that we call “mental mapping.” We 
convene groups of highly knowledgeable, experienced individuals, including researchers and 
practitioners, who are steeped in their respective fields but diverse in their perspectives and 
beliefs. We ask them to draw on their accumulated wisdom to make explicit their “mental maps” 
of what works to reach the outcome under consideration. Participants are asked to respond 
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initially to the question, “Considering the evidence from the research, theories, and experiences 
you have been exposed to over the years, what could a community most effectively do if it were 
committed to achieving the specific outcome under consideration (such as higher rates of School 
Readiness or higher rates of Family Economic Success)?” 

This process borrows from the thinking behind NIH Consensus Conferences, which 
provide “a vehicle for moving beyond the piecemeal presentation of evidence finm diverse 
bodies of literature and for ensuring the unbiased synthesis of findings that can inform broader 
discussions of effective strategies, in contrast to ‘up or down’ appraisals of individual 
programs.”^* 

PMI supplements the information that comes out of the mental mapping meetings by 
soliciting input from additional experts to fill in gaps. It also discards the claims around which 
there is little consensus and only weak support in theory or experience. We also do field-testing 
to make sure that the information we are conveying is readily understood, useful, and relevant. 

Using those methods, we have constructed a prototype Pathway to School Readiness. A 
second Pathway (to Family Economic Success) is currently under construction, and others will 
follow. 

Major Elements of Pathways 

The Pathways that appear on our website, www.PathwavsToOutcomes.ora are 
constructed to build a knowledge base that will contribute to efforts to improve outcomes among 
disadvantaged children, families, and neighborhoods by: 

• bridging disciplines; 

• identifying the interventions that contribute to achieving specified outcomes; 




13 



12 



• identifying easy-to-use indicators of progress toward specified outcomes; 

• identifying the attributes of actions that make them effective; and 

• identifying the elements of community and systems infrastructure that contribute to 
effectiveness. 

1. The Pathways approach bridges disciplines. 

We found it was possible, using the Pathways Mapping methodology, to cross 
disciplinary boundaries so we were no longer looking at single domains in isolation. PMI’s Map 
of the School Readiness Knowledge Base (Figure 1), for example, shows three domains we 
identified in which effective action is expected to lead to higher rates of school readiness: 
children’s health, their social and cognitive environments, and their neighborhood surroundings. 

All around the country, people tell each other that it takes more than family support 
services to strengthen families, more than child welfare services to keep children safe, more than 
the police to keep neighborhoods free of violence, and more than good preschool programs to get 
children ready for school. The National Academy of Sciences committee that produced From 
Neurons to Neighborhoods concluded that successful action to improve outcomes for children 
and families in adverse circumstances may have less to do with the impact of specific services or 
programs than with wide-ranging changes in housing, employment practices, community 
policing, and taxation. But there are few existing mechanisms that allow for such cross-cutting 
conclusions. Interventions and funding are designed as though each system could achieve its 
objectives on its own - even as though each individual program could, in isolation, achieve 
valued outcomes on its own. 

NRC (2000, p. 410). 

NRC (2000, pp. 339-340). 
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The mental mapping approach makes it possible to structure thinking broadly, across 
disciplinary domains and helping systems. Without having to invent the whole logic chain from 
scratch, users can see the many ways co mm unities can act effectively to improve important 
outcomes such as higher rates of school readiness. 

2. The Pathways approach can identify the interventions that contribute to achieving specified 
outcomes. 

We were able to identify specific actions (services, supports, and other interventions) that 
were likely to achieve the stipulated goals and outcomes. We organized these actions by the 
contributions they are hypothesized to make to achieving the outcome (see Appendix 2 for 
examples). Once we had identified the actions, we went back to the literature to add the 
rationale for believing the hypothesized connections between the actions and outcomes, and to 
identify the research evidence, where it exists, that supports the effectiveness of many of the 
actions. We did this because we believe that, while the information available to communities 
should not be limited to interventions that have been proved effective by methodologically 
elegant evaluations, it is important for users of that information to know which interventions are 
supported by formal effectiveness research. 

By focusing attention on the links between actions and outcomes, we believe we are 
bringing greater coherence and discipline to the design and implementation of interventions. But 
because we have been able to identify so many actions that contribute to the designated outcome, 
the question of choices and tradeoffs becomes acute. While we assume that every action listed 
will be useful, not every action listed is likely to make an equally powerful contribution. Surely 
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high-quality dental care is not as important as high-quality childcare. But is the prevention of 
domestic violence as important as the prevention of child abuse? 

We expect communities using Pathways to custom-craft a set of coherent priorities that 
take into account their unique circumstances, including what is already in place, and their own 
particular needs, resources, and opportunities. Pathways will help them apply a comprehensive 
lens to their assessment of needs and establish a framework for their analysis — recognizing that 
they can’t do everything, and certainly can’t do everything at once. 

3. The Pathways approach can identify easy-to-use indicators of progress toward specified 
outcomes. 

By making explicit the links between actions and outcomes. Pathways guides the search 
for sorely needed interim indicators of progress. As communities struggle to improve, reform, 
and expand services, supports, and infrastructure, and to introduce a greater emphasis on 
prevention, interim indicators (or benchmarks) become especially important for demonstrating 
short-term achievements - or the lack thereof - and for revealing the need for mi d-course 
corrections. 

Everyone who has worked at the front lines knows how long it takes before the 
achievement of long-term goals can be documented. We also know how difficult it is for 
community people or politicians to stay engaged with an initiative that will not reach its ultimate 
outcome for many years. Leaders have to know whether they’re on the right track, and that 
requires identifying interim milestones that can become reliable markers of progress. 

At Pathways, we believe that neither these interim indicators nor the indicators that 
measure outcomes should have to be produced individually and ad hoc by every local agency or 
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coalition. PMI has therefore sought to identify and describe a group of interim and outcome 
indicators with the following characteristics: 

• They are relatively easy and economical for communities to obtain 

• They reflect accurately the desired state or condition 

• They are readily understood by stakeholders across a community 

• They can be compared to data from other sources. 

As part of the School Readiness pathway, we collaborated with Child Trends to produce 
such indicators, each accompanied by a definition, a descriptions of its significance, and its 
source. These are now available for local people to select from and adapt to their own situation 
to document changes in conditions that matter. (See Appendix 3A and 3B for examples of 
indicators). 

4. The Pathways approach can identify attributes of the actions that make them effective. 

PMI’s mental mapping process may make its greatest contribution by identifying the 
attributes of effective interventions, the characteristics that make interventions effective. In 
addition to broadening the available information, by applying a comprehensive lens to “what 
works,” PMI makes possible a deeper level of understanding of how it works. Most research (on 
both human and neighborhood development) describes the natural course of normal or abnormal 
development. Few resources and little prestige flow to the intervention research that could guide 
people in what to do. Even fewer supports go to studies that could shed light onhow to do it. 

And yet, success in improving outcomes often depends less on the design of the 
intervention itself than on the characteristics of how the intervention is carried out, which tend to 
be hard to measure and are often neglected. Through our mental mapping process, we were able 



to identify the attributes of effectiveness that seem to characterize all effective interventions and 
also some of the specific attributes that characterize particular interventions (see Appendix 4 for 
examples). We learned that a host of efforts to improve children’s health, social environment, 
and neighborhood conditions change outcomes and are worth investing in when and if they 
incorporate certain attributes. 

The NAS Committee pointed out that what society knows about these factors is not 
empirical, although “the theoretical and experiential support [for these concepts] is strong.”^^ We 
were especially pleased to find that the NRC report surfaced a set of ingredients of success quite 
similar to ours, which include 

• sensitivity to cultural differences 

• the intangible sense of community and the message of social inclusion that 
communicates to children and families a sense of opportunity and high expectations 

• an interplay among the soundness of an intervention strategy, its acceptability to the 
intended recipients, and the quality of its implementation, and 

• the extent to which interventions are family-centered and community-based.^'* 

To cite an example of how this focus on attributes plays out, our early childhood mental 
mapping group didn’t spend time on whether “home visiting,” as a single category of 
intervention, works. The group surfaced information that is often hidden in more conventional 
summaries of research on “what works.” They concluded that in effective home visiting 
programs, staff training and supervision are designed to support practices that enable staff to 
make individuals and families feel respected and cared for, and staff are able to readily mobilize 
additional services and supports that go beyond the home visitor’s capacity. They cited 

NRC (2000, pp. 346, 353 and 366). 

NRC (2000, pp. 342, 346). 




experience indicating that in home visiting programs with disappointing results, a high 
proportion of mothers don’t engage with home visitors because they are depressed. Thus, our 
mental mappers hypothesized that in effective home visiting programs the home visitors would 
have the capacity to provide both these mothers and their children with the intensive and 
specialized help they need. 

To take another example, when we asked what interventions were likely to be most 
helpful in changing outcomes for the highest-risk, most disadvantaged families, the answers 
didn’t come in the form of reco mm endations for new programs. Rather, the answers came in the 
form of connections that must be built into existing programs and institutions — for example, to 
make it far easier and more routine for a Head Start teacher to obtain consultation from a mental 
health specialist, obtain a developmental assessment of a child she is concerned about, or 
mobilize housing assistance or substance abuse treatment for an overwhelmed mother. 

By spotlighting the attributes of effective services and supports, we can help solve one 
problem that has bedeviled efforts to scale up and spread model programs. On one hand, it is 
now part of the conventional wisdom that we caimot simply replicate or clone a successful 
model. We have to adapt it to meet local circumstances. But we also need to know that, in our 
adaptations, we are retaining the essential elements and are not inadvertently underrnining what 
works. One way to know this is by figuring out what, indeed, are those essential elements. If we 
can describe these with precision and in some depth, even in the absence of absolute certainty, 
our efforts to scale up will be much better informed and more effective, whether the initial 
determination of program effectiveness was made through randomized trials or by other means. 



5. The Pathways approach can identify the elements of community and systems infrastructure 
that contribute to effectiveness. 

Our mental mapping process confirmed how much context matters, and it enabled us to 
identify the elements of community and systems infrastructure that support and sustain effective 
change over time (see Appendix 5 for examples). 

We were able to identify the community contexts that don’t attach to a single program or 
intervention but are essential to improving outcomes. We concluded that communities are not 
likely to reach the results they are aiming for if individual programs are to be the sole unit of 
analysis. We cannot march program by program into the better future we seek. “What works” to 
achieve better outcomes for a community is a broader, more complicated question than “what 
programs work.” This deeper, more contextual level of analysis can clarify the limits of 
individual programs and illuminate the potential synergy in the work that occurs in several 
different domains. It also emphasizes the importance of strengthening community capacity to 
monitor programs and outcomes for an entire population - to scan what’s available and the gaps 
that remain. 

Many of the contextual elements identified by Pathways as contributing to improving 
outcomes do not fall under the clear responsibility of an existing institution or community entity. 
If they are to be responded to effectively, funders and policy makers must grapple with 
identifying who in the community can take responsibility and what new institutions or other 
mechanisms may need to be put in place. Those questions simply don’t surface when we ask 
only which programs are worth investing in. 

The policy and systems context can also be decisive. A hostile regulatory, funding, and 
accountability climate can seriously undermine what works. The structures and institutions 
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within which initiatives operate often destroy the very attributes that accounted for initially 
successful results, especially as they try to reach more people and make use of public funds. 
PMI’s findings suggest that when it comes to allocating resources we have to move beyond 
programmatic concerns, and beyond how much money is earmarked for particular programs, to 
focus on the terms on which money is made available. 

Funding issues are problematic today not simply because of the weak political support for 
helping troubled families and investing in poor neighborhoods. It is also because traditional 
funding practices are outmoded; they do not support new understandings of what makes for 
effective action. Public money often is bound by restrictions that directly undermine “what 
works.” And philanthropic funds tend to be severely time-limited, even when they seek 
outcomes that can only occur over the long term. 

As the Finance Project points out, “categorical funding streams that provide support only 
in response to narrowly defined problems and are available only when problems become chronic 
or severe” dominate the terrain, making it almost impossible to create multi-pronged, responsive, 
community-based support systems that would support effective interventions. And, of course, 
even categorical funds are rarely available at levels that match the needs being addressed. 

The importance of the funding, regulatory, and accountability context is most keenly 
recognized by practitioners. People who run successful programs are full of stories about their 
constant struggle to swim upstream, about how they have to be willing to break or bend the rules 
in order to get the job done. And they can do that - by stealth, while running pilot programs that 
remain small and operate at the margins and with special funding. But when they attempt to 
scale up, they are immediately confronted with rules, regulations, and funding realities that 
ultimately destroy or dilute the very attributes that made the original model successful. 




21 



20 



In the push to hold individual programs accountable for accomplishing valued outcomes, 
it is easy to forget that the conditions that determine their effectiveness are rarely under the 
control of those being held accountable. Nor are the levers that could change those conditions 
under local control. Those who are at the front lines of providing community services and 
supports know which rules and regulations interfere with effective operations, but the crafting of 
those mles and regulations are in the hands of others, people whom program leaders are unlikely 
to be able to influence. 

Unless we’re prepared to rely forever on wizards who can beat the bureaucracies and the 
dysfunctional regulations and funding practices - because they are some combination of Mother 
Teresa, Machiavelli, and a CPA - we have to pay more attention to the context, and how it must 
be changed. By identifying the elements of the policy and systems context that support what 
works, and by encouraging the formation of alliances that could bring about needed change, we 
can ensure that many more talented people and mobilized communities are able to act on what 
we know to change outcomes for large populations of children and families. 

The Need for a Stronger Knowledge Base 

We hope that Pathways will help all stakeholders - the public, funders, practitioners, and 
those who benefit directly from effective services, supports, and systems - address the challenge 
of working in an era when trying hard and good intentions are no longer good enough. We hope 
to make it easier for local communities at any stage of development to work more effectively and 
coherently. Local decision-making, and the harnessing of local initiative, imagination, and 
adaptation can be pre-eminent, but are much more likely to lead to effective action when 



Hayes (2002). 
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combined with a broad collection of accessible knowledge that is drawn from practice and theory 
as well as research. 

We hope that our website’s interactive capacity will enable us to maintain Pathways as an 
evolving resource, one that becomes an ever more useful summary of what is currently known in 
each of the Pathways domains. As the website’s users distill local experience and feed it back to 
us, we expect to continuously modify the knowledge base to integrate and reflect the 
accumulating experience and research. 

In the future, we also expect to include guidance to communities on how to use the 
information more strategically. We hope to do that without destroying the comprehensiveness, 
inclusiveness, and sense of interconnections among components that users have identified as one 
of Pathways’ strengths. We hope that the Pathways information and approach will not only 
provide useful tools for community assessment and for program design and improvement but 
also encourage stakeholders to think in a more integrated, ambitious way about building the 
knowledge needed to achieve important outcomes. 

Part of the knowledge-building task certainly involves improving society’s ability to 
assess the impact of complex individual programs. But to make significant progress on the 
social problems that currently loom as so intractable, we must get beyond the program-by- 
program approach that has dominated the last several decades. As Public/Private Venture’s Gary 
Walker and Jean Baldwin Grossman suggest in their influential paper, “Philanthropy and 
Outcomes: Dilemmas in the Quest for Accountability,” the general assumption that dominated 



In “Philanthropy and Outcomes: Dilemmas in the Quest for Accountability,” Walker and Baldwin Grossman (1999), 
suggest that impact assessments of individual programs should probably be limited to initiatives that are stable and have 
a track record, “have refined their substantive strategies based on years of operational experience,” and represent “solid 
implementation of a particular substantive idea.” Data from these sources could then be used to develop “a set of 
operational benchmarks by which to assess the progress of newer operational manifestations” of those particular 
exemplars. 
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incrementalist social policy in the late 1900s was that “useful changes in policy and practice 
would come in the form of additions or modest alterations to extant progrartiming - in short, we 
were just a tinker away from resolving the problem.”^^ If instead the challenge is to formulate a 
much bolder agenda, the PMI approach may suggest some useful directions. Because the 
Pathways dig deep enough to reach the issues that remain hidden in traditional approaches to 
determining “what works,” they have the potential to stimulate action on the areas that Walker 
and Grossman suggest may now be paramount. These include: 

• encouraging serious rethinking of basic substantive strategies; 

• illuminating the need for new or expanded institutions, as opposed to additional 
“programs”; 

• emphasizing the importance of building capacity to reach outcomes at sufficient scale to 
matter; and 

• making visible the importance of filling gaps among interventions and forging 
connections among them. 

The knowledge base that emerges from the Pathways process is arguably wider, deeper, 
more coherent, and more actionable than most similar collections of information. But will the 
products of this more inclusive mindset be credible? We believe that credibility established by 
generalizing from a preponderance of rigorously analyzed evidence is essential for societies 
attempting to design urgently needed social strategies. Once we agree that there is knowledge 
that is worth having and analyzing even though it is not certain knowledge, we are in a better 
position to incorporate the lessons to be learned from both the successful and unsuccessful 
intervention efforts of the last two decades. We will be able to modernize our old vocabulary. 



” Walker (1999). 




24 



23 



our traditional, compartmentalized ways of thinking, and our institutional arrangements to meet 
changing needs, and to address the new complexities of how to make what works actually work. 

Of course this approach to knowledge building implies a trade-off It suggests that the 
knowledge-building process should shift from predominant reliance on a knowledge base 
consisting of a few narrow, circumscribed change efforts - even when that knowledge is certain, 
final, and “true.” Instead, our knowledge-building efforts should enable the people making daily 
decisions about policies and programs to make more useful decisions by providing them with 
access to knowledge that is integrated, coherent, accessible and deep, and deals with a broad 
range of promising change efforts - even if some of that knowledge is tentative, contingent, and 
approximate. 

The time may be ripe for a conceptual shift in our analytic work - a shift that would 
honor multiple ways of knowing, allow us to address more fundamental questions, and 
encourage us to assemble useful data about the effects of promising interventions even in the 
absence of absolute proof There is increasing interest in a more inclusive paradigm for 
knowledge building - a paradigm that doesn’t give up on searching out the patterns that connect 
activities to results, even in the absence of certainty. Far from being “unscientific,” a paradigm 
that combines theory, logic, and evidence with intelligent judgment, analogy, insight, and 
creativity is the combination that the Nobel prize-winning biologist David Baltimore identifies as 
the basis for "the most fundamental progress in science."^* This less linear approach to 
systematic learning could begin to produce the practical knowledge needed to improve outcomes 
while simultaneously combating the prevailing nihilism, which holds that nothing can be known 
because the certainty we demand is not attainable. 



Baltimore (1997) 
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A sense of the potential of going to scale with a more pragmatic approach comes from the 
extraordinary work now going on in England. Soon after he was elected, Prime Minister Tony 
Blair established the Health Development Agency (HDA) to synthesize “evidence, theory, and 
practical knowledge and wisdom” to eliminate income-based health inequalities in the nation. 

The HDA is currently engaged in a massive effort to collect and appraise evidence from a wide 
variety of sources, “acknowledging the legitimacy of different types of learning.” It is 
converting that evidence into advice and guidance aimed at bringing about the change in both 
practice and systems that is needed to reduce or eliminate income disparities in health-related 

29 

outcomes. 

At a time of little enthusiasm in this country for governmental action to alleviate poverty 
and promote social justice, a new, more generous approach to identifying “what works” could 
provide the underpinning for an action agenda bold enough to make some of our higher 
aspirations into realistic goals. It could become a significant step toward achieving a wider 
distribution of family stability and economic security, universal school readiness and school 
success, and other goals the nation values. 



” Kelly and Speller (2003) 
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FIGURE 1 



Overview of the Pathway to School Readiness 
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OUTCOME: CHILDREN ARE READY FOR SCHOOL 
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APPENDIX 1 



CHARACTERISTICS OF EFFECTIVE INTERVENTIONS 
that make them difficult to evaluate with prevailing methods 



Intervention has multiple, interactive components, often from diverse domains 

Intervention is constantly evolving in response to experience and changing conditions 

Implementers "believe in,” are actively committed to and engaged with the intervention 

Intervention/program design is adapted to local circumstances, preferences, needs and 
strengths 



Intervention emphasize significant front-line flexibility within established quality parameters 

Intake/recruitment into the program is under local control (often dependent on individuals’ 
choices) within broad parameters 



Program responds to individuals in family, peer, & neighborhood context 



Effectiveness is heavily dependent on community, policy, institutional, regulatory, and 
resource context 



Interventions emphasizes continuing, respectful, and trusting relationships, and other hard- 
to-measure attributes 



Desired outcomes are likely to be community-wide 
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APPENDIX 2 



EXAMPLES OF ACTIONS THAT CONTRIBUTE TO SCHOOL READINESS 

Goal: I. Good Health > Subgoal: lA. Healthy, wanted births > 

Action Area: lAl. Intended, well-timed pregnancies 

Actions: Services and Supports 

Provide reproduction-related health education in a variety of forms appropriate to various 
actual and potential users. 

Provide a wide range of family planning services and methods that are effective, affordable, 
and acceptable to a variety of actual and potential users; give clients information that helps 
them choose and use with satisfaction contraception and other means of ensuring that 
pregnancies are intended. 

Provide a range of counseling and mental health services for women facing unplanned, 
unwanted pregnancies. 

Provide community options for minors seeking family planning services or related guidance. 

Actions: Community Interventions 

Promote positive alternatives to early childbearing (aimed at postponing both first and 
subsequent pregnancies), taking full account of lessons learned in relevant fields, including 
youth development, school reform, and youth engagement. 

Use community-wide public education campaigns to emphasize the importance of: 

All births being intended and wanted 

Postponing pregnancies until parents are ready for parenthood 
Discouraging early, unmarried childbearing 
Women being in good health before becoming pregnant 
Recognizing early signs of pregnancy 

Understanding the responsibilities involved in parenting and meeting the needs of 
dependent children 

Offer a variety of reproductive services in settings, locations, and times that make them 
conveniently accessible. 

Minimize barriers to the use of family planning services and reproduction-related health 
education, through outreach and other means. 

Shape co mmuni ty norms in ways that discourage early childbearing and promote the idea 
that pregnancy should be intended and wanted. 
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Create an environment that attracts providers of reproductive health care to the community. 

Actions: Policy and System Supports 

Miriiniize financial harriers to reproductive health services hy simplifying health insurance 
eligibility and enrollment processes, expanding eligibility for affordable health insurance, 
and advocating for health insurance to cover a variety of family planning and other 
reproductive services. 

Make funding available in amounts and on terms that ensure a variety of family planning 
and other reproductive services are offered in settings, locations, and times that make them 
acceptable, accessible, and appropriate. 

Goal: II. Supportive Social and Cognitive Environments > Sub-goal: IIA. Strong bonds with 
primary caretaker and supportive home > 

Action Area: IIAl. Responsive caretaker, supportive home 

Actions: Services and Supports 

Provide formal and informal services and supports that help all new parents (and 
especially high-risk families) develop responsive, nurturing parent-child relationships and 
strong, respectful, trusting relationships with support systems. Such services and supports 
include: 

Well-trained, supervised adults who offer support during pregnancy, childbirth, 
and child’s early life (e.g., Doulas, home visitors) 

Education (coaching, mentoring, exposure to models of good parenting) that helps 
parents improve their skills, understand attachment issues, and hold realistic 
expectations for children’s development 
Family support services and centers 

“Warm” lines that offer information and support before crises occur 
Adequate time and other supports for breastfeeding (in hospitals, among health 
and childcare professionals, among employers) 

Make special efforts to reach out to, serve, and support the highest risk families through 
culturally sensitive home visits, peer support, and family support and by establishing 
connections with family support centers, health and mental health programs, child care 
and preschool programs, and specialized programs. 

Encourage and help family service providers to serve as coaches, mentors, and models of 
good parenting. 

Actions: Community Activities 

Shape community-wide norms in ways that: 

Promote stable family life 
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Value support from formal sources, including home visits, family support 

programs, and parent education 

Promote breastfeeding and responsible child rearing 

Value time spent parenting 

Provide abundant opportunities to strengthen parents in their child-rearing role and to 
build strong relationships between young children and their parents and other adult 
caregivers. 

Help to coimect formal services and agencies with neighborhood networks, to maximize 
the extent to which individual families experience services and agencies as responsive to 
them and as “on their side.” 

Encourage supportive adults (spouses or other partners, kin, neighbors, and informal 
groups) to participate actively in child rearing; encourage father involvement with young 
children 

Provide resource materials about positive parenting at community events. 

Actions: Policy and System Supports 

Advocate for supports to parents that help them balance labor force participation with 
good parenting. In particular, work to ensure that employers and the mles governing 
Temporary Assistance for Needy Families (TANF) take into accoxmt the importance of 
allowing adequate time for parenting, especially during a child’s first year. 

Advocate for policies that provide paid parental leave to all workers with very young 
children by: 

Extending protections of the Family and Medical Leave Act to workers in mid- 
size and, eventually, small businesses 

Providing income to workers who take family leave, especially for infants, 
through a funding pool that combines public finds with contributions from 
employers and employees 

Extending parental leave from three months to six months 
Allowing the use of unemployment insurance to fund parental leave 

Goal: III. Safe, Strong Neighborhoods > Sub-goal: IIIA. Supportive neighborhoods > 

Action Area: IIIAl. Strong social connections, engaged residents 

Actions: Services and Supports 

Promote social, cultural, and civic activities for neighborhood residents to help them form 
and sustain relationships and build networks. 

Create social activities and parenting classes for pre-school parents and children at 
libraries, community centers, and places of worship. 



Distribute neighborhood directories of services and numbers to call for assistance, 
resident block lists, and other resources that connect residents to each other and services. 

Create neighborhood conversation groups where residents can share hopes, dreams, and 
priorities for their children and the neighborhood. 

Form block groups and neighborhood watch programs. 

Conduct a “walkers and talkers” door-to-door campaign to help neighbors get to know 
each other and learn about resources. 

Actions: Community Activities 

Partner to host cultural and family events, festivals, and other social activities that 
encourage neighbors to know and support each other. 

Collaborate across neighborhood cultural and ethnic groups, associations, agencies, 
businesses, and places of worship to work for neighborhood improvement. 

Offer courses in leadership to help residents advocate for better housing and 
neighborhood conditions. 

Strengthen the capacity of neighborhood associations to be vehicles for resident networks 
and to take action on issues. 

Hold events that engage residents with each other, such as parties, barbecues, coffees, 
and garage sales. 

Organize candidate forums and briefings to inform residents of pending legislation and to 
create opportunities for residents to share their priorities. 

Hold voter registration and “get the vote out” events. 

Use community action and community organizing to help build capacity within 
neighborhood associations. 

Actions: Policy and System Supports 

Require agencies and governing bodies to seek and obtain community approval for 
community redevelopment and family relocation plans. 

Encourage agencies and governing bodies to provide support to neighborhood 
associations in order to involve residents in neighborhood planning and development. 
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APPENDIX 3A 



EXAMPLES OF INDICATORS THAT MEASURE PROGRESS TOWARD SUB-GOALS 
Goal: I, Good Health > Subgoal: lA, Healthy, wanted hirths > 

• Percent low birthweight births 

• Percent births to teens 

• Percent women receiving late or no prenatal care 

Goal: H, Supportive Social and Cognitive Environments > Sub-goal: HA. Strong bonds with 
primary caretaker and supportive home > 

• Percent of children whose biological/adoptive parents are legally married 

• Percent of parents who read to their children in the past week 

• Percent of children who, on weeknights during the school year, usually go to bed at 
the same time each night 

Goal: III, Safe, Strong Neighborhoods > Sub-goal: HIA, Supportive neighborhoods > 

• Home ownership: Percent who rent, own or have another arrangement 

• Percent of children who have moved more than once in the past year 
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APPENDIX 3B 



EXAMPLES OF INDICATORS THAT MEASURE PROGRESS TOWARD 
OUTCOME OF SCHOOL READINESS 

Physical and Motor Development 

• Percent parents reporting their children are in excellent, good, poor etc. health 

• Percent of children in expected height and weight range for their age 

• Percent of children receiving required immunizations by the time of school entry 

• Percent of children demonstrating fine motor skills 

• Percent of children with newly detected hearing problems at school entry 

• Percent of children with newly detected vision problems at school entry 

• Percent of children with newly detected asthma problems at school entry 

• Percent of children with newly detected disability/developmental problems requiring 
special education services 

Social and Emotional Development 

• Percent of children who easily join others in play 

• Percent of children who make and keep friends easily 

Language Development 

• Percent of children proficient in understanding, interpreting a story read to him/ her 

• Percent of children able to comprehend directions, as described by interviewers 

• Percent of children with satisfactory scores on a measure of receptive vocabulary 

Cognitive Development 

• Percent of children able to count the total number of objects in a set 

• Percent of children proficient at sorting, classifying, comparing math materials by 
various rules of attributes 

Disposition to Learn 

• Percent of children working independently almost always, occasionally, etc. 

• Percent of children who can follow directions 

• Percent of children showing eagerness to learn new things, never, occasionally, 
regularly, etc. 

• Percent of children who persist in completing tasks never, occasionally, regularly, etc 
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APPENDIX 4 



EXAMPLES OF ATTRIBUTES OF EFFECTIVE SERVICES AND SUPPORTS 

Initiatives aimed at improving the health of young children, their social and cognitive 
environments, and the neighborhoods they live in are likely to be effective when and if certain 
conditions, including the following, are met 

• They are of high quality, in that staff are technically competent and not burdened with 
case loads that interfere with providing quality services; they are responsive to clients 
with diverse needs, backgrounds and languages, and characterized by mutually respectful 
interactions 

• Their staffs have the capacity, training, support, and supervision that ensure high quality 
services; staff have continuing access to training, supervision, and consultation that help 
them acquire necessary knowledge and skills and develop a rich repertoire of responses 
to unexpected circumstances. 

• They are psychologically and geographically accessible, in that they are welcoming to 
families, and respectful of their diverse needs. Children and fa mi lies have reason to trust 
the institutions and individuals offering services and support. 

• They are able to connect families with basic supports and services, supportive networks, 
and specialized services for specific conditions, such as mental illness, substance abuse, 
and disturbed parent-child relationships 

• They ensure that the duration, frequency, and intensity of interventions, services, and 
supports are carefully calibrated to the needs, resources, and risk factors of specific 
children and families, and of the population served by the program. 

• They pay special attention to uptake, participation, and attrition in order to reach and 
persevere with the highest-risk individuals and families 

• Their staff working with children have skills, support, and time to be sensitive to the 
needs of their families, and staff working with families and other adults have skills, 
support, and time to be sensitive to the needs of their children 



APPENDIX 5 



EXAMPLES OF COMMUNITY AND SYSTEMS CONTEXTS 
THAT CONTRIBUTE TO EFFECTIVENESS 

Examples of supportive community infrastructure 

Community outcomes are most likely to improve when and if there are community entities that 
support children and families and the institutions serving them by developing such capacities as 
the following; 

Community entities take continuing responsibility for monitoring formal agencies and 
organizations to ensure that all appropriate services and supports are available to those who 
need them. 

The community formulates and disseminates consistent messages in efforts to shape norms 
about the importance of supporting families in their childrearing responsibilities 

The community is committed to assuring the quality and availability of such primary services 
as health care, child care, housing, training in life/work skills, income and employment 
supports, family support, parent education, literacy development, safe parks, church and 
volunteer affiliations, and recreational and cultural opportunities 

Community groups work together to assure multiple entry points to essential services and 
supports. 

Co mm unities partner with allies at the local, state, regional, and national levels to maximize 
the chances of sustaining “what works” over time. 

The community recognizes and acts on the need for neighborhood-based networks and 
services that help families keep children safe 

The community’s residents know and socialize with their neighbors, and look to each other 
for support 

Examples of supportive policies and systems 

Community outcomes are most likely to improve when and if supportive policies and systems 
are in place, including the following: 

Funding policies recognize the importance of strengthening the capacity of service providers 
and community organizations by providing core funding for activities that cannot be 
supported as parts of categorical projects or programs. 
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Adequate, stable, predictable funding is available for activities aimed at preventing problems, 
in addition to services that respond to identified problems. 

Funding is sufficiently flexible that services and supports can be tailored to the needs of 
individual families and communities. Funding policies allow programs to use multiple 
funding streams for two- and three-generation services and other combinations of services 
that traditionally have been funded separately and categorically. 

Policies support the provision of multiple entry points to essential services and supports. 

Policies, regulations, payment mechanisms, and training practices approach the provision of 
preventive services, and services to respond to acute and chronic crises in ways that: 

• Impose minimal burdens on providers and families 

• Support the provision of comprehensive, continuing, appropriate, and acceptable 
care 

• Encourage the development of respectful, trusting relationships between providers 
and families 

Make assistance with problems an integral part of programs that engage and support families 
in their everyday lives, such as parent support groups, child care, or classes in English as a 
Second Language, fitness and life skills 
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